Matthias Bannert

• current occupation: data scientist / software developer @ETH Zurich
• occasional consultant
• studied economics @UniKN, PhD @ETHZ: partly economics, mostly methodology + stats
• CTO of Swiss startup fanpictor from 2012-2014
• open source software projects: timeseriesdb, tstools, dropR, RAdwords

Approach

# listen - forget
# see - remember
# do - understand

Goals

• plan
• apply
• scale

• Day 1: Organize
  • Introduction
  • Data Generating Processes
  • Types of Data
  • Manage and Archive
• Day 2: Process and Communicate
  • Visualization
  • Methodology

You Draw It! (NY Times)

• download automatically
• read spreadsheet
• process
• visualize

• create report
• dynamic figures & tables
• html, pdf, docx

Quelle: all Logos taken from their respective companies' website.

"Premature optimization is the root of all evil."

• First appeared in 1993
• designed by Ihaka and Gentleman
• Last Stable Release: 3.4.1

Why R?

• interpreted language
• interfaces to many compiled languages
• easy to learn
• open source, license cost free
• backed by Microsoft
• one-of-a-kind ecosystem, wide range of packages

• R Interpreter
• CRAN Repository /w 10K+ packages
• CRAN Task Views
• Community: stackoverflow, Mailinglist, …
• R Studio IDE / R Studio Server

• Switch to LTR Layout
• Console vs. Scripting window
• comments
• short cut cmd+enter: run selection
• short cut ctrl+1, ctrl+2: switch windows
• short cut ctrl+L: clear console screen
• short cut command+D: multiple cursors @instances
• file explorer
• plot window
• .Rproj

• vector
• matrix
• data.frame
• list

• [row,col]: Index
• {}: function or loop body
• (): function parameters

• ls()
• rm()
• c()
• matrix()
• data.frame()
• list()

getting help: ?function name

• head()
• tail()
• str()
• function()
• lapply()
• data()

getting help: ?function name

• i_am_a_snake

• jeSuisUnCamel

1. How many observations does the dataset mtcars have?
2. What's the miles-per-gallon average, median?
3. Which is the most ecological car?
4. Which is the most ecological car by cylinders?
5. How is mpg distributed?
6. Why does solving analytics exercises through programming make sense?

• scripting language is good start
• understanding a language helps to remember syntax
• many tasks can be solved w/o database, larger stack
• programming makes tasks scalable and reproducible

Why?

• Form Expectations (Proposals / Grants)
• Detectability
• Analyzing complex systems

study: http://ieeexplore.ieee.org/document/1622367/?reload=true

• sources: Webservers, IoT devices
• event based files
• not aggregated, large amounts of data

solutions:

• specific tools: awstats
• SaaS products
• programming

source tagesanzeiger.ch

• user specific logging
• data similar to log data
• e-commerce tracking
• often received through APIs in software as-a-service (Saas) products
  • Google Analytics
  • Google Adwords
  • Adsense

source: http://gantalcala.org/

• input: online forms, smartphone apps, interviews, paper
• result in cross sectional or panel data

source: pinterest

• web scraping
• scraping spreadsheets, html tables
• image processing
• prices from car vendor
• text mining

source: youtube, spaceballs

• REST popular on the web
• SOAP a bit more old fashioned, but used in B2B often
• APIs are an easy choices when receiving third party data
  • can be automatically processes
  • conflict with security policies

Discuss in smaller groups:

• Does using scripts to access the web conflict with your department's policies?
• What would be a good option to receive 3rd party data?
• How could 3rd party data be merged with internal data?
• Where do you stand on cloud computing?

ts2 <- ts(rnorm(20),
          start = c(1995,1),
          frequency = 4)
ts2

##            Qtr1       Qtr2       Qtr3       Qtr4
## 1995 -0.3625870 -1.3589722  2.9884704 -0.3030327
## 1996  1.3504988  0.2517712 -1.3133845 -0.5131971
## 1997 -0.4809161 -0.9930176  0.2278636  0.2560794
## 1998 -0.4265592 -0.2902753 -1.8648031 -1.4235040
## 1999 -0.3613658 -0.4917637  1.4510953 -1.1790192

examples: monthly revenues over time, stocks, aggregated log files

head(mtcars)

##                    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

• multiple variables
• one period

• multiple variables
• longitudinal data
• e.g. German Socio-Economic Panel (GSOEP)

l <- list()
l$element1 <- 2
l$element2 <- head(mtcars,4)
l

## $element1
## [1] 2
## 
## $element2
##                 mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4      21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag  21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710     22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1

examples: meta information, sector classification (hierarchical), GDP components, translations, attributes, properties

Represent each of the following dataset types using R:

• time series
• cross section
• panel
• a nested structure.

Please suggest an in-memory representation and a file based representation.

• character
• numeric
• factor (be warned of stringsAsFactor = T)
• …